{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 精确率与召回率，RoC曲线与PR曲线\n",
    "\n",
    "### 一、 TP, FP, TN, FN\n",
    "\n",
    "**P（Positive）：**代表1\n",
    "\n",
    "**N（Negative）：**代表0\n",
    "\n",
    "**T（True）：**代表预测正确\n",
    "\n",
    "**F（False）：**代表预测错误\n",
    "\n",
    "![image-20200910151725931](https://gitee.com/Little_Six/repository_pic/raw/master/null/image-20200910151725931.png)\n",
    "\n",
    "**TP：**预测正样本，预测正确\n",
    "\n",
    "**FP：**预测正样本，预测错误\n",
    "\n",
    "**FN：**预测负样本，预测错误\n",
    "\n",
    "**TN：**预测负样本，预测正确\n",
    "\n",
    "### 二、精确率、召回率、ROC曲线\n",
    "\n",
    "准确率：$P = \\frac{(TP+TN)}{(TP+TN+FP+FN) }$\n",
    "\n",
    "精确率：$P = \\frac{(TP)}{(TP+FP) }$   预测正样本，预测正确/预测的所有正样本个数\n",
    "\n",
    "召回率：$R = \\frac{(TP)}{(TP+FN) }$   预测正样本，预测正确/实际的所有正样本个数\n",
    "\n",
    "F1：$\\frac{2}{F_1} = \\frac{1}{P} + \\frac{1}{R}$\n",
    "\n",
    "\n",
    "\n",
    "灵敏度：  $TPR = \\frac{(TP)}{(TP+FN) }$   = 召回率     真阳率\n",
    "\n",
    "1-特异度：$FPR = \\frac{FP}{FP + TN }$                        假阳率   预测正样本，预测错误/实际的所有的负样本个数\n",
    "\n",
    "\n",
    "\n",
    "ROC曲线说的是$TPR$和$FPR$的曲线，P-R曲线说的就是精确率与召回率的曲线：\n",
    "\n",
    "![image-20200910155116443](https://gitee.com/Little_Six/repository_pic/raw/master/null/image-20200910155116443.png)\n",
    "\n",
    "### 三、代码练习\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAOLklEQVR4nO3df6jd9X3H8eeryVwZUzuWW7BJbCyL0FsZUy7OUlgtuhGFJv+4koB0HWLabnZ/WAaODi32r7VsQiFbGzbpWvBX+0e9lJRAO8UhjfOKqTWRjLvUmqsybzunf0irsvf+OMdyuDn3nu9Nzr0n95PnAwLne84n576/uTdPv37POfmmqpAkbXzvmvQAkqTxMOiS1AiDLkmNMOiS1AiDLkmN2DypL7xly5basWPHpL68JG1ITz311M+ramrYYxML+o4dO5ibm5vUl5ekDSnJz5Z7zFMuktQIgy5JjTDoktQIgy5JjTDoktSIkUFPcm+SV5I8u8zjSfLVJPNJnkly1fjHlCSN0uUI/RvArhUevwHY2f+1H/insx9LkrRaI9+HXlWPJdmxwpI9wDer9+/wHknyniSXVNXLY5pRWjf3PfECDx99cdJjqHHT77uIuz7+obE/7zjOoW8FTg1sL/TvO02S/UnmkswtLi6O4UtL4/Xw0Rc5/vLrkx5DOiPj+KRohtw39KoZVXUQOAgwMzPjlTV0Tpq+5CIe/PSHJz2GtGrjOEJfALYPbG8DXhrD80qSVmEcQZ8FPtl/t8s1wGueP5ek9TfylEuS+4FrgS1JFoC7gN8AqKqvAYeAG4F54A3gz9dqWEnS8rq8y2XfiMcL+MuxTSRJOiN+UlSSGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGtEp6El2JTmRZD7JHUMevzTJI0meTvJMkhvHP6okaSUjg55kE3AAuAGYBvYlmV6y7G+Bh6rqSmAv8I/jHlSStLIuR+hXA/NVdbKq3gQeAPYsWVPARf3bFwMvjW9ESVIXXYK+FTg1sL3Qv2/QF4GbkywAh4DPDXuiJPuTzCWZW1xcPINxJUnL6RL0DLmvlmzvA75RVduAG4FvJTntuavqYFXNVNXM1NTU6qeVJC2rS9AXgO0D29s4/ZTKLcBDAFX1I+DdwJZxDChJ6qZL0J8Edia5LMkF9F70nF2y5gXgOoAkH6QXdM+pSNI6Ghn0qnobuA04DDxH790sx5LcnWR3f9nngVuT/Bi4H/hUVS09LSNJWkObuyyqqkP0XuwcvO/OgdvHgY+MdzRJ0mr4SVFJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGdAp6kl1JTiSZT3LHMms+keR4kmNJ7hvvmJKkUTaPWpBkE3AA+GNgAXgyyWxVHR9YsxP4G+AjVfVqkveu1cCSpOG6HKFfDcxX1cmqehN4ANizZM2twIGqehWgql4Z75iSpFG6BH0rcGpge6F/36DLgcuTPJ7kSJJdw54oyf4kc0nmFhcXz2xiSdJQXYKeIffVku3NwE7gWmAf8M9J3nPab6o6WFUzVTUzNTW12lklSSvoEvQFYPvA9jbgpSFrHq6qt6rqp8AJeoGXJK2TLkF/EtiZ5LIkFwB7gdkla74LfAwgyRZ6p2BOjnNQSdLKRga9qt4GbgMOA88BD1XVsSR3J9ndX3YY+EWS48AjwF9X1S/WamhJ0ulGvm0RoKoOAYeW3HfnwO0Cbu//kiRNgJ8UlaRGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGdPrXFs8l9z3xAg8ffXHSY6hRx19+nelLLpr0GNIZ2XBH6A8ffZHjL78+6THUqOlLLmLPHyy9ZK60MWy4I3To/aV78NMfnvQYknRO2XBH6JKk4Qy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIzoFPcmuJCeSzCe5Y4V1NyWpJDPjG1GS1MXIoCfZBBwAbgCmgX1JpoesuxD4K+CJcQ8pSRqtyxH61cB8VZ2sqjeBB4A9Q9Z9Cfgy8MsxzidJ6qhL0LcCpwa2F/r3/VqSK4HtVfW9lZ4oyf4kc0nmFhcXVz2sJGl5XYKeIffVrx9M3gXcA3x+1BNV1cGqmqmqmampqe5TSpJG6hL0BWD7wPY24KWB7QuBK4BHkzwPXAPM+sKoJK2vLkF/EtiZ5LIkFwB7gdl3Hqyq16pqS1XtqKodwBFgd1XNrcnEkqShRga9qt4GbgMOA88BD1XVsSR3J9m91gNKkrrZ3GVRVR0CDi25785l1l579mNJklbLT4pKUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1olPQk+xKciLJfJI7hjx+e5LjSZ5J8sMk7x//qJKklYwMepJNwAHgBmAa2Jdkesmyp4GZqvp94DvAl8c9qCRpZV2O0K8G5qvqZFW9CTwA7BlcUFWPVNUb/c0jwLbxjilJGqVL0LcCpwa2F/r3LecW4PvDHkiyP8lckrnFxcXuU0qSRuoS9Ay5r4YuTG4GZoCvDHu8qg5W1UxVzUxNTXWfUpI00uYOaxaA7QPb24CXli5Kcj3wBeCjVfWr8YwnSeqqyxH6k8DOJJcluQDYC8wOLkhyJfB1YHdVvTL+MSVJo4wMelW9DdwGHAaeAx6qqmNJ7k6yu7/sK8BvA99OcjTJ7DJPJ0laI11OuVBVh4BDS+67c+D29WOeS5K0Sn5SVJIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIa0SnoSXYlOZFkPskdQx7/zSQP9h9/IsmOcQ8qSVrZyKAn2QQcAG4ApoF9SaaXLLsFeLWqfg+4B/i7cQ8qSVpZlyP0q4H5qjpZVW8CDwB7lqzZA/xr//Z3gOuSZHxjSpJG2dxhzVbg1MD2AvCHy62pqreTvAb8LvDzwUVJ9gP7AS699NIzGnj6fRed0e+TpNZ1CfqwI+06gzVU1UHgIMDMzMxpj3dx18c/dCa/TZKa1+WUywKwfWB7G/DScmuSbAYuBv5nHANKkrrpEvQngZ1JLktyAbAXmF2yZhb4s/7tm4B/q6ozOgKXJJ2Zkadc+ufEbwMOA5uAe6vqWJK7gbmqmgX+BfhWknl6R+Z713JoSdLpupxDp6oOAYeW3HfnwO1fAn863tEkSavhJ0UlqREGXZIaYdAlqREGXZIakUm9uzDJIvCzM/ztW1jyKdTzgPt8fnCfzw9ns8/vr6qpYQ9MLOhnI8lcVc1Meo715D6fH9zn88Na7bOnXCSpEQZdkhqxUYN+cNIDTID7fH5wn88Pa7LPG/IcuiTpdBv1CF2StIRBl6RGnNNBPx8vTt1hn29PcjzJM0l+mOT9k5hznEbt88C6m5JUkg3/Frcu+5zkE/3v9bEk9633jOPW4Wf70iSPJHm6//N94yTmHJck9yZ5JcmzyzyeJF/t/3k8k+Sqs/6iVXVO/qL3T/X+F/AB4ALgx8D0kjV/AXytf3sv8OCk516Hff4Y8Fv92589H/a5v+5C4DHgCDAz6bnX4fu8E3ga+J3+9nsnPfc67PNB4LP929PA85Oe+yz3+Y+Aq4Bnl3n8RuD79K74dg3wxNl+zXP5CP18vDj1yH2uqkeq6o3+5hF6V5DayLp8nwG+BHwZ+OV6DrdGuuzzrcCBqnoVoKpeWecZx63LPhfwzkWDL+b0K6NtKFX1GCtfuW0P8M3qOQK8J8klZ/M1z+WgD7s49dbl1lTV28A7F6feqLrs86Bb6P0XfiMbuc9JrgS2V9X31nOwNdTl+3w5cHmSx5McSbJr3aZbG132+YvAzUkW6F1/4XPrM9rErPbv+0idLnAxIWO7OPUG0nl/ktwMzAAfXdOJ1t6K+5zkXcA9wKfWa6B10OX7vJneaZdr6f1f2L8nuaKq/neNZ1srXfZ5H/CNqvr7JB+mdxW0K6rq/9Z+vIkYe7/O5SP08/Hi1F32mSTXA18AdlfVr9ZptrUyap8vBK4AHk3yPL1zjbMb/IXRrj/bD1fVW1X1U+AEvcBvVF32+RbgIYCq+hHwbnr/iFWrOv19X41zOejn48WpR+5z//TD1+nFfKOfV4UR+1xVr1XVlqraUVU76L1usLuq5iYz7lh0+dn+Lr0XwEmyhd4pmJPrOuV4ddnnF4DrAJJ8kF7QF9d1yvU1C3yy/26Xa4DXqurls3rGSb8SPOJV4huB/6T36vgX+vfdTe8vNPS+4d8G5oH/AD4w6ZnXYZ9/APw3cLT/a3bSM6/1Pi9Z+ygb/F0uHb/PAf4BOA78BNg76ZnXYZ+ngcfpvQPmKPAnk575LPf3fuBl4C16R+O3AJ8BPjPwPT7Q//P4yTh+rv3ovyQ14lw+5SJJWgWDLkmNMOiS1AiDLkmNMOiS1AiDLkmNMOiS1Ij/B+Za5o638Zo2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "from sklearn.metrics import auc \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "y = np.array([1, 1, 2, 2])  \n",
    "scores = np.array([0.1, 0.4, 0.35, 0.8])  \n",
    "\n",
    "fpr, tpr, thresholds = metrics.roc_curve(y, scores, pos_label=2)\n",
    "plt.plot(fpr, tpr)\n",
    "metrics.auc(fpr, tpr) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxP9f7A8dd7xjKWyU52su8jo5SSbCkZbnWjREpEoUi7rlS3q9USKiG5KfVToi7XUqRkz07ZY0S2wWCGWd6/P86ZuV9jlu8w3/nOzPf9fDzmMd+zv8/5Lu9zPp9zPh9RVYwxxgSuIH8HYIwxxr8sERhjTICzRGCMMQHOEoExxgQ4SwTGGBPgLBEYY0yAs0SQB4hIDxFZ6O84/E1EqojIGREJzsZtVhMRFZF82bVNXxKRrSLS+jKWy7OfQRFpLSKR/o7DlywRZDER2SciMe4P0mERmSYiRX25TVWdoaodfLmNnMg91u2ShlV1v6oWVdUEf8blL25Cqnkl61DVBqq6NIPtXJL8AvUzmFdYIvCNzqpaFAgDmgLP+zmey+LPs9y8coadGXa8jb9YIvAhVT0MLMBJCACISEEReVtE9ovIXyLygYgU8pjeRUQ2iMhpEdktIh3d8cVEZIqIHBKRgyLyWlIRiIj0FpGf3dcfiMjbnnGIyBwRGeq+riAiX4nIURHZKyKDPeZ7WURmicinInIa6J1yn9w4prvL/yEiw0UkyCOO5SLynoicEpHfRKRtimXT24flIjJaRE4AL4tIDRH5QUSOi8gxEZkhIsXd+f8NVAG+da++nkl5pioiS0XkVXe90SKyUERKe8TTy92H4yLyUsorjBT7XUhE3nHnPyUiP3u+b0AP9z09JiIveix3nYisEJGT7n6PF5ECHtNVRB4XkZ3ATnfcWBE54H4G1onIzR7zB4vIC+5nI9qdXllElrmzbHSPRzd3/jvdz9NJEflFRBp7rGufiDwrIpuAsyKSz/MYuLGvdeP4S0TedRdN2tZJd1s3eH4G3WUbiMgiETnhLvtCGsc1ze+DG9tKj/dzgDhFVyHu8P+Jc9V9SkSWiUgDj/VOE5GJIjLfjXG5iFwtImNEJMr9bDZNcSyeF5Ft7vSPk7aTSsxpfodyLVW1vyz8A/YB7dzXlYDNwFiP6WOAuUBJIBT4FviXO+064BTQHidJVwTqutO+AT4EigBlgdXAo+603sDP7utWwAFA3OESQAxQwV3nOuAfQAHgGmAPcJs778tAHNDVnbdQKvs3HZjjxl4N2AH08YgjHhgC5Ae6uftT0st9iAcGAfmAQkBN91gUBMrg/ACNSe1Yu8PVAAXyucNLgd1AbXd9S4FR7rT6wBngJvdYvO3ue7s03tcJ7vIVgWDgRjeupG1+5G6jCXAeqOcu1wxo4e5TNWA78KTHehVYhPN5KOSOewAo5S7zFHAYCHGnPY3zmaoDiLu9Uh7rqumx7muBI8D1bswPusesoMfx2wBU9th28jEFVgA93ddFgRapHedUPoOhwCE39hB3+Po0jmt634cg9z1/GagFRAFNPZZ92F2moLueDR7TpgHH3OMfAvwA7AV6ucfiNWBJis/SFvdYlASWA6+501oDkR4xpfkdyq1/fg8gr/25H6gzQLT7ZfkeKO5OE+AsUMNj/huAve7rD4HRqayzHM6PSyGPcfclfZBTfAkF2A+0cof7Aj+4r68H9qdY9/PAx+7rl4Fl6exbsBtHfY9xjwJLPeL4EzcJueNWAz293If9aW3bnacrsD7Fsc4oEQz3mP4Y8F/39T+Azz2mFQYukEoicL/8MUCTVKYlbbNSin3unsY+PAnM9hhWoE0G+x2VtG3gd6BLGvOlTATvA6+mmOd34BaP4/dwKp/fpESwDBgJlE5jn9NKBPd5vk/p7Fe63wePbZ3ASaDPp7Ou4m5MxdzhacBHHtMHAds9hhsBJ1Psd3+P4TuA3e7r1vwvEaT7Hcqtf1Yu6BtdVXWxiNwCfAaUBk7inNUWBtaJSNK8gvMDC87ZyLxU1lcV5wz7kMdyQThn/hdRVRWRmThfxmXA/cCnHuupICInPRYJBn7yGL5knR5K45wF/eEx7g+cs+QkB9X9dnhMr+DlPly0bREpC4wDbsY58wvC+VHMjMMer8/hnNnixpS8PVU9JyLH01hHaZyzyt2Z3Y6I1AbeBcJx3vt8OGeUnlLu91PAI26MClzlxgDOZyS9ODxVBR4UkUEe4wq460112yn0AV4BfhORvcBIVf3Oi+16G2NG3wdUdZ+ILMH5YZ6QPJNTpPhP4O/uehLdSaVxrkIB/vLYVkwqwylv4vA8Fkmf25S8+Q7lOlZH4EOq+iPOmUlSmf0xnA9gA1Ut7v4VU6diGZwPYo1UVnUA52y6tMdyV6lqg1TmBfgcuEdEquKcwXzlsZ69HusorqqhqnqHZ9jp7NIxnOKTqh7jqgAHPYYrise32p3+p5f7kHLb/3LHNVbVq3CKTCSd+TPjEE7RHeDUAeAUx6TmGBBL6u9NRt4HfgNqufvwAhfvA3jsh1sf8CxwL1BCVYvj/LAlLZPWZyQ1B4B/pni/C6vq56ltOyVV3amq9+EU470BzBKRIuktk8kYM/o+ICJ34FwlfA+85bHs/UAXoB1QDOfKAS49tplR2eN10uc2JW++Q7mOJQLfGwO0F5EwVU3EKUse7Z7tIiIVReQ2d94pwEMi0lZEgtxpdVX1ELAQeEdErnKn1XCvOC6hquuBo8BkYIGqJp29rAZOu5VwhdyKx4Yi0tybHVHntswvgX+KSKibaIbyvysOcH40BotIfhH5O1APmJfZfXCF4hSznRSRijjl457+wimjvRyzgM4icqM4lbcjSeNHxH3fpgLvuhWFwW4FaUEvthMKnAbOiEhdYIAX88fjvH/5ROQfOFcESSYDr4pILXE0FpGkBJbyeHwE9BeR6915i4hIJxEJ9SJuROQBESnj7n/SZyjBjS2RtI/9d8DVIvKkWxkcKiLXp5wpo++DOBX7U3Cujh7Eeb+SfnBDcU4sjuNcVbzuzT5l4HERqSQiJXES9hepzHNF36GcyhKBj6nqUZwK1pfcUc8Cu4CV4tyZsxin4g9VXQ08BIzGOQv8kf+dfffCuazfhlM8Mgson86mP8c5W/rMI5YEoDPOXUx7cc7IJuOcUXlrEE657h7gZ3f9Uz2mr8Kp2DuGc+l+j6omFblkdh9G4lR4ngL+A3ydYvq/gOHi3BEzLBP7gKpudfdlJs7VQTROxer5NBYZhlNJuwanzPoNvPv+DMM5e43G+dFL7cfF0wJgPk4l/B84VyKeRRbv4iTjhTgJZgpOJTU4dTyfuMfjXlVdi1NHNB7neO8ilTvB0tER2CoiZ4CxOPUesap6Due9Xe5uq4XnQqoajVPJ3xmnyGwncGsa20jz+wBMAuao6jz3M9QHmOwmvunu8TmI83lamYn9SstnOMd1j/v3WsoZsug7lOMk3VlizBUTkd7AI6p6k79jySxxHvo7iVOEs9ff8ZjsJSL7cD67i/0diz/YFYEJWCLSWUQKu+Xeb+Oc8e/zb1TGZD9LBCaQdcGpEPwTpziru9olsglAVjRkjDEBzq4IjDEmwOW6B8pKly6t1apV83cYxhiTq6xbt+6YqpZJbVquSwTVqlVj7dq1/g7DGGNyFRH5I61pVjRkjDEBzhKBMcYEOEsExhgT4CwRGGNMgLNEYIwxAc5niUBEporIERHZksZ0EZFxIrJLRDaJyLW+isUYY0zafHlFMA2n9cK03I7zWH8toB9Ou+3GGGOymc+eI1DVZSJSLZ1ZugDT3bZdVopIcREp77Zbn/W+7gR7U+v8yxhjcrbV+ysSki+exhX+gqeyvlkgf9YRVOTidtYjubjLw2Qi0k9E1orI2qNHj17e1iwJGGNyGVV45rv23PBeHx6c2ZW4BN/8ZPvzyeLUeoNKNdWp6iScTioIDw+/snTog2xqjDG+IABHFsGyFXR44G4SBo8nvw+2489EEMnFfYRWIvU+Qo0xJmCcPBnLnj1RXHut03nfyJGt6d69YfKwL/izaGgu0Mu9e6gFcMpn9QPGGJMLzJnzG/XrTyAi4nNOnYoFoFCh/D5NAuDDKwIR+RxoDZQWkUhgBDhXNar6ATAPuAOnv9JzOH31GmNMwDly5CyDB8/niy+2AtCiRSVOnoylWLGQbNm+L+8aui+D6Qo87qvtG2NMTqeqzJixmSee+C8nTsRQuHB+Xn+9DQMHXkdwcPYV2OS6ZqiNMSavGDDgP3z44ToA2rW7hkmT7qR69RLZHoc1MWGMMX7StWtdihcPYcqUCBYufMAvSQDsisAYY7LNzp3H+f77vfTvHw5Ax4412bfviWyrC0iLJQJjjPGx+PhE3n13BSNGLOX8+XjCwq6mRYtKAH5PAmCJwBhjfGrjxsP06TOXdeucu+N79WpCrVol/RzVxSwRGGOMD5w/H89rry1j1KjlxMcnUqVKMT788E46dqzp79AuYYnAGGN84Pnnv2f06JUAPP54c/71r7aEhhb0c1Sps0RgjDE+8MwzLVmxIpI332zHzTdX9Xc46bLbR40xJgssWrSbu+/+kvj4RACuvroov/zycI5PAmCJwBhjrkhUVAx9+syhQ4dP+frr7Xz88frkaSKpNbKc81jRkDHGXKbZs7fz2GPzOHz4DAULBjNixC307h3m77AyzRKBMcZk0uHDZxg0aD6zZm0D4MYbKzNlSgR165b2c2SXxxKBMcZk0pw5vzFr1jaKFMnPqFHteOyx5gQF5Y5ioNRYIjDGGC/ExsYTEuL8ZPbt24w9e6IYMKA51aoV93NkV84qi40xJh2Jicr48aupXn0sf/xxEoCgIOGNN9rniSQAlgiMMSZNv/9+jFatPmbQoPkcPnyGzz/f4u+QfMKKhowxJoW4uATefvsXRo78kfPnEyhXrggTJ3birrvq+Ts0n7BEYIwxHrZsOUKvXrNZv/4wAA89FMY773SgRIlCfo7MdywRGGOMh8REZfPmI1StWoxJkzrToUMNf4fkc5YIjDEBb+vWI9SvXwYRoXHjcsyZ051WrapStGgBf4eWLayy2BgTsKKjzzNw4DwaNnyfr77anjz+jjtqBUwSALsiMMYEqAULdtGv33fs33+KfPmC2LfvpL9D8htLBMaYgHLiRAxDhixg+vSNAFx7bXmmTIkgLOxqP0fmP5YIjDEBY8OGw3Ts+Cl//XWWggWDGTmyNU89dSP58gV2KbklAmNMwKhduxRFixagdu1STJ4cQe3apfwdUo5gicAYk2epKp99tpnOnetw1VUFKVw4P0uX9qZChdBc3UhcVgvs6yFjTJ61b99JbrvtUx54YDbPPbc4eXylSldZEkjBrgiMMXlKQkIiEyeu4fnnv+fs2ThKlizEjTdW9ndYOZolAmNMnrF9+1H69JnLihWRANx7bwPee+92ypYt4ufIcjZLBMaYPGHv3ijCwj7kwoUEypcvysSJnejata6/w8oVLBEYY/KE6tVL8Pe/1yckJB9vv92B4sVD/B1SruHTymIR6Sgiv4vILhF5LpXpVURkiYisF5FNInKHL+MxxuQdMTFxPP/8YlavPpg87pNPujJ5coQlgUzyWSIQkWBgAnA7UB+4T0Tqp5htOPClqjYFugMTfRWPMSbv+OmnPwgL+5BRo5bTr9+3JCYqAMHBdiPk5fDlUbsO2KWqe1T1AjAT6JJiHgWucl8XA/70YTzGmFzu9OnzPP74f2jVaho7dhynfv0yfPDBnXY76BXyZR1BReCAx3AkcH2KeV4GForIIKAI0C61FYlIP6AfQJUqVbI8UGNMzjdv3k769/+OAwdOky9fEC+8cBMvvHAzBQtaVeeV8uUVQWopWlMM3wdMU9VKwB3Av0XkkphUdZKqhqtqeJkyZXwQqjEmJzt1KpYePb7mwIHThIdXYN26fowceaslgSziy6MYCXg+xVGJS4t++gAdAVR1hYiEAKWBIz6MyxiTC6gqqhAUJBQrFsK4cR3566+zPPlki4BvJC6r+fJorgFqiUh1ESmAUxk8N8U8+4G2ACJSDwgBjvowJmNMLvDnn9H87W9fMHr0iuRxPXs2YdgwaynUF3x2RFU1HhgILAC249wdtFVEXhGRCHe2p4C+IrIR+Bzoraopi4+MMQFCVZky5Vfq15/AnDm/89ZbvxATE+fvsPI8nxawqeo8YF6Kcf/weL0NaOnLGIwxucOePVH07fstP/ywF4BOnWrxwQd3UqhQfj9HlvdZTYsxxq8SEhIZN24VL774AzEx8ZQuXZhx4zrSvXtDROy20OxgicAY43ezZm0nJiae++5ryNixHSlTxhqJy06WCIwx2e7ChQSio89TqlRhgoODmDIlgp07j9O5cx1/hxaQrPrdGJOt1qw5SHj4JHr2nE3SvSF165a2JOBHdkVgjMkW587FMWLEEt59dyWJicq5c3EcOXKWcuWK+ju0gGeJwBjjc0uX7qNv32/ZtesEQUHCsGE3MHLkrRQubHcE5QSWCIwxPqOqDB48n/Hj1wDQqFFZpkyJoHnzin6OzHiyRGCM8RkR4aqrCpI/fxDDh7fiueduokCBYH+HZVKwRGCMyVLHjp1j9+4TXH99JQBeeukWevRoTP361mBkTuXVXUMiUkBEavo6GGNM7qWqzJy5hXr1JtC16xdERcUAEBKSz5JADpdhIhCRTsBmYJE7HCYis30dmDEm94iMPE2XLjO5776vOHbsHPXrl+HcOWsjKLfwpmjoFZwOZZYAqOoGuzowxgAkJiqTJ//K008v4vTp81x1VUHeeacDffo0teYhchFvEkGcqp5M8aZaC6HGGPr0mcu0aRsAiIiow8SJd1Cx4lUZLGVyGm/qCLaLyL1AkNu3wBhgpY/jMsbkAg880IiyZYswc+bdfPNNN0sCuZQ3iWAg0AxIBL4GYoEnfBmUMSZn2rLlCGPH/u88sG3ba9izZzDdullLobmZN0VDt6nqs8CzSSNE5C6cpGCMCQDnz8fzr3/9zOuv/0RcXCLh4RVo2bIKAEWKFPBzdOZKeXNFMDyVcS9mdSDGmJxp1apImjWbxMiRPxIXl8iAAeE0alTO32GZLJTmFYGI3IbTsXxFEXnXY9JVOMVExpg87OzZC7z00hLGjFmJKtSqVZLJkyNo1aqqv0MzWSy9oqEjwBacOoGtHuOjged8GZQxxv9efPEHxo5dRVCQ8PTTN/Dyy62t28g8Ks1EoKrrgfUiMkNVY7MxJmNMDvDiizezefMR3nijHeHhFfwdjvEhb+oIKorITBHZJCI7kv58HpkxJlvNnfs7d9wxg7i4BADKlCnC99/3siQQALxJBNOAjwEBbge+BGb6MCZjTDY6cuQs3bvPokuXmcyfv4tPPtno75BMNvMmERRW1QUAqrpbVYcDt/o2LGOMr6kqn366iXr1JvDFF1spXDg/Y8d25KGHwvwdmslm3jxHcF6cJ0V2i0h/4CBQ1rdhGWN8af/+U/Tv/x3z5+8CoF27a5g06U6qVy/h58iMP3iTCIYARYHBwD+BYsDDvgzKGONbCxfuZv78XRQvHsK773agd+8wezI4gGWYCFR1lfsyGugJICKVfBmUMSbrnT17Ifkp4D59mnLw4Gn69WtG+fKhfo7M+Fu6dQQi0lxEuopIaXe4gYhMxxqdMybXiI9P5M03l1O16hj27IkCnC4kR4xobUnAAOkkAhH5FzAD6AH8V0RexOmTYCNQO3vCM8ZciY0bD3P99ZN59tnFHD8ewzff/ObvkEwOlF7RUBegiarGiEhJ4E93+PfsCc0Yc7nOn4/ntdeWMWrUcuLjE6lSpRiTJt3JbbdZn1LmUuklglhVjQFQ1RMi8pslAWNyvvXrD9Gjx9ds334MERg4sDmvv96W0NCC/g7N5FDpJYJrRCSpqWkBqnkMo6p3ZbRyEekIjAWCgcmqOiqVee4FXsbp9Wyjqt7vffjGmJQKFszH7t1R1KlTismTI7jppir+DsnkcOklgrtTDI/PzIpFJBiYALQHIoE1IjJXVbd5zFMLeB5oqapRImLPJxhzGX799RBNm16NiFC/fhnmz+/BjTdWJiTEmzvETaBLr9G5769w3dcBu1R1D4CIzMSpd9jmMU9fYIKqRrnbPHKF2zQmoERFxTBs2EKmTt3A55/fTffuDQFo06a6nyMzuYkvTxcqAgc8hiOB61PMUxtARJbjFB+9rKr/TbkiEekH9AOoUsUuc40BmD17O489No/Dh89QsGAwx4+f83dIJpfyZSJI7TFFTWX7tYDWQCXgJxFpqKonL1pIdRIwCSA8PDzlOowJKIcPn2HQoPnMmuVcXLdsWZnJkyOoW7e0nyMzuZXXiUBECqrq+UysOxKo7DFcCecW1JTzrFTVOGCviPyOkxjWZGI7xgSMdev+pH37fxMVFUuRIvkZNaodjz3WnKAgax7CXL4MWx8VketEZDOw0x1uIiLvebHuNUAtEakuIgWA7sDcFPN8g9uSqfv0cm1gTybiNyag1K9fhjJlinDbbTXYuvUxBg68zpKAuWLeNEM9DrgTOA6gqhvxohlqVY0HBgILgO3Al6q6VUReEZEId7YFwHER2Ybz1PLTqno887thTN6UmKhMmrSOkyedTgILFcrPsmW9mT+/B1WrFvdzdCav8KZoKEhV/0jRMmGCNytX1XnAvBTj/uHxWoGh7p8xxsPvvx/jkUe+5eef97NmzUE++sg5fypXrqifIzN5jTeJ4ICIXAeo+2zAIMC6qjTGR+LiEnjnnRW8/PJSzp9P4Oqri3L77bX8HZbJw7xJBANwioeqAH8Bi91xxpgstn79Ifr0mcv69YcBeOihMN55pwMlShTyc2QmL/MmEcSranefR2JMgNu9+wTXXTeZ+PhEqlUrzqRJd9K+fQ1/h2UCgDeJYI17W+cXwNeqGu3jmIwJSDVqlKRnz8aEhhbgn/9sS9GiBfwdkgkQGd41pKo1gNeAZsBmEflGROwKwZgrdObMBQYPns+KFf97AH/KlAjGjr3dkoDJVt7cPoqq/qKqg4FrgdM4HdYYYy7TggW7aNBgIu+9t5r+/f+DcwMd1m+w8YsMi4ZEpChOY3HdgXrAHOBGH8dlTJ504kQMQ4YsYPr0jQA0a1aeKVMiLAEYv/KmjmAL8C3wpqr+5ON4jMmzZs3axuOPz+PIkbOEhORj5MjWDB16A/nyeXVhbozPeJMIrlHVRJ9HYkwedvJkLP36fUtUVCytWlXlo486U7t2KX+HZQyQTiIQkXdU9SngKxG5pMVPb3ooMyaQqSqJiUpwcBDFi4cwcWInoqJiePTRcGsfyOQo6V0RfOH+z1TPZMYY2LfvJP36fUubNtV57rmbAJI7jTEmp0mzcFJVV7sv66nq955/OJXGxpgUEhISGTduFQ0bTmTRoj2MH7+a2Nh4f4dlTLq8qaV6OJVxfbI6EGNyu+3bj9Kq1TSeeOK/nD0bR/fuDfn110et32CT46VXR9AN55bR6iLytcekUOBk6ksZE3ji4xN5442feeWVZVy4kECFCqG8/34nIiLq+Ds0Y7yS3qnKapw+CCoBEzzGRwPrfRmUMblJUJCwcOEeLlxIoG/fa3nzzfYULx7i77CM8VqaiUBV9wJ7cVobNcZ4iImJIzr6AmXLFiEoSJg8uTMHDpymTZvq/g7NmExLs45ARH50/0eJyAmPvygROZF9IRqTsyxb9gdNmnzAAw98ndw0RK1apSwJmFwrvaKhpO4oS2dHIMbkdKdPn+f55xczceJaAPLnD+bYsXOUKVPEz5EZc2XSu3006WniykCwqiYANwCPAvbJNwFl/vydNGw4kYkT15IvXxAjRtzCr7/2syRg8gRv7mv7BmguIjWA6cB/gM9wOrQ3Jk9TVfr2/ZYpU5z7I8LDKzB1agSNGpXzc2TGZB1vniNIVNU44C5gjKoOAir6NixjcgYRoVKlqwgJycfbb7dnxYo+lgRMnuNVV5Ui8negJ9DVHZffdyEZ419//hnN7t0nuPnmqgC88MLN9OzZmBo1Svo5MmN8w9sni2/FaYZ6j4hUBz73bVjGZD9VZcqUX6lffwJ33/0lx4+fA6BAgWBLAiZPy/CKQFW3iMhgoKaI1AV2qeo/fR+aMdlnz54o+vb9lh9+2AvAnXfWJi7OWl83gcGbHspuBv4NHAQEuFpEeqrqcl8HZ4yvJTUSN3z4Es6di6N06cKMG9eR7t0bWq9hJmB4U0cwGrhDVbcBiEg9nMQQ7svAjMkOvXp9w2efbQbg/vsbMWbMbXZLqAk43tQRFEhKAgCquh0o4LuQjMk+ffteS6VKVzF3bndmzLjLkoAJSN5cEfwqIh/iXAUA9MAanTO51Jo1B/nhh708+6zTWUzr1tXYtWsQBQtaU9EmcHnz6e8PDAaewakjWAa858ugjMlq587FMWLEEt59dyWJicqNN1ZOvj3UkoAJdOl+A0SkEVADmK2qb2ZPSMZkraVL9/HII3PZvTuKoCBh2LAbaNasgr/DMibHSK9jmhdweiL7FaeJiVdUdWq2RWbMFTp1KpZnnlnEpEm/AtCoUVmmTImgeXN7MN4YT+lVFvcAGqvq34HmwIDMrlxEOorI7yKyS0SeS2e+e0RERcTuRDJZ5qWXljBp0q/kzx/EK6+0Zu3afpYEjElFekVD51X1LICqHhURb+4wSiYiwTg9m7UHIoE1IjLX8w4kd75QnDqIVZmK3JhUqGry/f//+Mct7N17klGj2tKgQVk/R2ZMzpXej/s1IvK1+zcbqOEx/HU6yyW5Ducp5D2qegGYCXRJZb5XgTeB2ExHb4xLVfnss820aTOdCxcSAChdujDffnufJQFjMpDeFcHdKYbHZ3LdFYEDHsORwPWeM4hIU6Cyqn4nIsPSWpGI9AP6AVSpUiWTYZi8LjLyNAMG/IfvvtsBwIwZm3jooaZ+jsqY3CO9Pou/v8J1p/Z8viZPdIqaRgO9M1qRqk4CJgGEh4drBrObAJGYqHz00TqefnoR0dEXKFasIO+804HevcP8HZoxuYovb6COxOndLEkl4E+P4VCgIbDULdO9GpgrIhGqutaHcZk8YNeuE/Tt+y1Ll+4DoEuXOkyc2IkKFUL9G5gxuZAvE8EaoJbbbPVBoDtwf+n/fVkAABv8SURBVNJEVT2FR3/IIrIUGGZJwHjjp5/+YOnSfZQtW4Tx42/nnnvqWyNxxlwmrxOBiBRU1fPezq+q8SIyEFgABANTVXWriLwCrFXVuZkP1wSykydjKV48BIDevcM4evQcffo0pVSpwn6OzJjcLcNbQkXkOhHZDOx0h5uIiFdNTKjqPFWtrao1kvowUNV/pJYEVLW1XQ2Y1Jw/H8+IEUuoWnUMO3ceB5wuJJ95pqUlAWOygDdXBONwOqr/BkBVN4rIrT6NyhjXypWR9Okzl23bjgKwYMFuatUq5eeojMlbvEkEQar6R4ry1wQfxWMMAGfPXuCll5YwZsxKVKFWrZJMmRKR3FCcMSbreJMIDojIdYC6TwsPAnb4NiwTyFatiuT++79mz54ogoOFYcNuZMSIWyhUKL+/QzMmT/ImEQzAKR6qAvwFLOYy2h0yxlvFi4dw8OBpmjQpx5QpEdZSqDE+5k3n9Udwbv00xmd+/nk/LVtWRkSoU6c0P/zwIM2bVyB//mB/h2ZMnudN5/Uf4fFEcBJV7eeTiExAOXLkLIMHz+eLL7byySdd6dWrCQA33lg5gyWNMVnFm6KhxR6vQ4C/cXEbQsZkmqoyY8Zmnnjiv5w4EUPhwvmTG4szxmQvb4qGvvAcFpF/A4t8FpHJ8/bvP0X//t8xf/4uANq3v4ZJkzpTrVpxP0dmTGC6nCYmqgN2D5+5LKtWRdKu3b85c+YCxYuHMHr0bTz4YBNrHsIYP/KmjiCK/9URBAEngDR7GzMmPWFhV1O58lXUrVuaCRPuoHx5ayTOGH/LqPN6AZrgNBoHkKiq1gy08Vp8fCLjx6+mV68mlCxZiIIF87F8+cOUKFHI36EZY1zptjXk/ujPVtUE98+SgPHaxo2Huf76yQwZsoChQxckj7ckYEzO4k0/xKtF5FqfR2LyjNjYeIYP/4Hw8I/49ddDVKlSjPvua+jvsIwxaUizaEhE8qlqPHAT0FdEdgNncXoeU1W15GAu8csvB+jTZy6//XYMERg4sDmvv96W0NCC/g7NGJOG9OoIVgPXAl2zKRaTy+3adYKbb/6YxESlTp1STJkSQcuW1se0MTldeolAAFR1dzbFYnK5mjVL0q/ftZQsWYiXXrqFkBBfdoBnjMkq6X1Ty4jI0LQmquq7PojH5CJRUTE89dRCHnooLLl56IkTO9kzAcbkMuklgmCgKO6VgTGevv56O48/Po/Dh8+wbt0hNmx4FBGxJGBMLpReIjikqq9kWyQmVzh8+AwDB87jq6+2A3DTTVWYPLmzJQBjcrEM6wiMAaeRuOnTNzJkyAKiomIpWrQAb7zRjv79wwkKso+KMblZeomgbbZFYXK8kydjeeqphURFxdKxY00++KATVataI3HG5AVpJgJVPZGdgZicJzFRSUxU8uULokSJQnz44Z2cOxfHAw80tqIgY/IQb54sNgHot9+O0arVx4wa9XPyuLvvrk/PntZSqDF5jSUCc5G4uARef/0nmjT5gOXLDzBlynpiY+P9HZYxxofsiR+TbP36Qzz88Fw2bDgMQJ8+TXnrrfb2YJgxeZx9ww1xcQmMGLGUN99cTkKCUq1acT76qDPt2l3j79CMMdnAEoEhX74gVq06SGKi8sQT1/Paa20oWrSAv8MyxmQTSwQBKjr6PNHRF6hQIRQRYfLkzhw+fIYbbqjs79CMMdnMKosD0IIFu2jY8H169PiapL6GqlcvYUnAmABliSCAHD9+jgcf/IaOHWewf/8poqPPc/x4jL/DMsb4mU8TgYh0FJHfRWSXiFzS4b2IDBWRbSKySUS+F5GqvownUKkqs2Zto379iUyfvpGQkHy8+WY7Vq58hNKlC/s7PGOMn/msjkBEgoEJQHsgElgjInNVdZvHbOuBcFU9JyIDgDeBbr6KKRCpKj16fM3nn28BoFWrqnz0UWdq1y7l58iMMTmFL68IrgN2qeoeVb0AzAS6eM6gqktU9Zw7uBKo5MN4ApKIUL9+GUJDC/D++51YsuRBSwLGmIv48q6hisABj+FI4Pp05u8DzE9tgoj0A/oBVKliXR9mZO/eKPbsiaJtW+c5gGefbUnv3mFUqnSVnyMzxuREvrwiSK1BGk11RpEHgHDgrdSmq+okVQ1X1fAyZcpkYYh5S0JCImPHrqRhw/fp1m0WR46cBSB//mBLAsaYNPnyiiAS8LwfsRLwZ8qZRKQd8CJwi6qe92E8edq2bUd55JG5rFgRCUBERB3rJ8AY4xVfJoI1QC0RqQ4cBLoD93vOICJNgQ+Bjqp6xIex5FlxcQm88cZyXn11GRcuJFChQijvv9+JiIg6/g7NGJNL+CwRqGq8iAwEFuD0fzxVVbeKyCvAWlWdi1MUVBT4P7dp4/2qGuGrmPKi++//mlmznBux+va9lrfeak+xYiF+jsoYk5v4tIkJVZ0HzEsx7h8er9v5cvuB4IknrmfDhsN8+OGdtGlT3d/hGGNyIXuyOJf58cd9jBy5NHn4ppuqsH3745YEjDGXzRqdyyVOnz7Ps88u4oMP1gFw663VadXKeRA7Xz7L58aYy2eJIBeYN28njz76HZGRp8mfP4gXX7yZFi3s2TtjTNawRJCDHTt2jief/C8zZmwG4LrrKjJlSgQNG5b1c2TGmLzEEkEO9sorPzJjxmYKFcrHa6+14Yknric42IqBjDFZyxJBDqOquLfSMnJka/766yyvv96GGjVK+jkyY0xeZaeXOYSq8tFH67jxxqnExsYDUKJEIb744h5LAsYYn7JEkAPs3n2Ctm2n06/fd6xcGcmXX271d0jGmABiRUN+5DQSt4rhw38gJiaeMmUK8957t3PvvQ38HZoxJoBYIvCTrVuP8PDDc1m9+iAAPXo0YsyYjtZjmDEm21ki8JP16w+zevVBKlYM5cMP76RTp9r+DskYE6AsEWSjo0fPUqZMEcC5Ajh5MpaePRtbI3HGGL+yyuJscO5cHMOGLaRatbFs334UcLqQHDjwOksCxhi/sysCH1uyZC99+37L7t1RBAUJy5b9Qb161suaMSbnsETgI6dOxfLMM4uYNOlXABo1KsvUqV0ID6/g58iMMeZilgh84Oef99O9+ywOHowmf/4gXnqpFc8+exMFCgT7OzRjjLmEJQIfuPrqohw/HkOLFpWYPLkzDRpYI3HGmJzLEkEWUFUWLdpD+/bXICLUrFmSn39+iLCwq62ROGNMjme/UlfowIFTdO78Obfd9ikff7wheXyzZhUsCRhjcgW7IrhMiYlOI3FPP72I6OgLFCtWkIIFrQ7AGJP7WCK4DDt3Hqdv32/58cc/AOjatS4TJtxBhQqhfo7MGGMyzxJBJv3yywHatp1ObGw8ZcsWYfz427nnnvrJfQgYkyQuLo7IyEhiY2P9HYoJICEhIVSqVIn8+fN7vYwlgkwKD69ArVoladq0PO++24FSpayROJO6yMhIQkNDqVatmp0omGyhqhw/fpzIyEiqV6/u9XJWm5mB8+fj+ec/l3Hs2DkAChQIZvnyh/nkk66WBEy6YmNjKVWqlCUBk21EhFKlSmX6KtSuCNKxcmUkffrMZdu2o2zffoxPP70LgNDQgn6OzOQWlgRMdrucz5wlglScPXuB4cN/YOzYVahC7dqlePTRZv4OyxhjfMKKhlL4/vs9NGr0PmPGrCIoSHjuuZZs3Nifm2+u6u/QjMm04OBgwsLCaNiwIZ07d+bkyZPJ07Zu3UqbNm2oXbs2tWrV4tVXX0VVk6fPnz+f8PBw6tWrR926dRk2bJg/diFd69ev55FHHvF3GGk6fvw4t956K0WLFmXgwIFpznfixAnat29PrVq1aN++PVFRUYBT5j948GBq1qxJ48aN+fVXp+2yo0eP0rFjxyyL0xKBhx07jtO+/b/Zu/ckYWFXs3p1X/71r3aEhNiFk8mdChUqxIYNG9iyZQslS5ZkwoQJAMTExBAREcFzzz3Hjh072LhxI7/88gsTJ04EYMuWLQwcOJBPP/2U7du3s2XLFq655posjS0+Pv6K1/H6668zaNCgbN1mZoSEhPDqq6/y9ttvpzvfqFGjaNu2LTt37qRt27aMGjUKcJLxzp072blzJ5MmTWLAgAEAlClThvLly7N8+fIsidN+4TzUrl2KJ564njJlivD00zeSP789IGayyDs+qit4SjOex3XDDTewadMmAD777DNatmxJhw4dAChcuDDjx4+ndevWPP7447z55pu8+OKL1K1bF4B8+fLx2GOPXbLOM2fOMGjQINauXYuIMGLECO6++26KFi3KmTNnAJg1axbfffcd06ZNo3fv3pQsWZL169cTFhbG7Nmz2bBhA8WLFwegZs2aLF++nKCgIPr378/+/fsBGDNmDC1btrxo29HR0WzatIkmTZoAsHr1ap588kliYmIoVKgQH3/8MXXq1GHatGn85z//ITY2lrNnz/LDDz/w1ltv8eWXX3L+/Hn+9re/MXLkSAC6du3KgQMHiI2N5YknnqBfv35eH9/UFClShJtuuoldu3alO9+cOXNYunQpAA8++CCtW7fmjTfeYM6cOfTq1QsRoUWLFpw8eZJDhw5Rvnx5unbtyowZMy45LpcjoBPBX3+dYfDg/9K/fzNuvdW51Wr06Ky73DImp0hISOD777+nT58+gFMs1KzZxfVeNWrU4MyZM5w+fZotW7bw1FNPZbjeV199lWLFirF582aA5CKN9OzYsYPFixcTHBxMYmIis2fP5qGHHmLVqlVUq1aNcuXKcf/99zNkyBBuuukm9u/fz2233cb27dsvWs/atWtp2LBh8nDdunVZtmwZ+fLlY/Hixbzwwgt89dVXAKxYsYJNmzZRsmRJFi5cyM6dO1m9ejWqSkREBMuWLaNVq1ZMnTqVkiVLEhMTQ/Pmzbn77rspVarURdsdMmQIS5YsuWS/unfvznPPPZfh/qfmr7/+onz58gCUL1+eI0eOAHDw4EEqV66cPF+lSpU4ePAg5cuXJzw8nOHDh1/W9lIKyESgqnz66SaefHIBJ07E8Pvvx1i//lG7w8P4TibO3LNSTEwMYWFh7Nu3j2bNmtG+fXvA+Q6k9XnPzPdg8eLFzJw5M3m4RIkSGS7z97//neBg52q7W7duvPLKKzz00EPMnDmTbt26Ja9327ZtycucPn2a6OhoQkP/9/T+oUOHKFPmf508nTp1igcffJCdO3ciIsTFxSVPa9++PSVLlgRg4cKFLFy4kKZNmwLOVc3OnTtp1aoV48aNY/bs2QAcOHCAnTt3XpIIRo8e7d3ByQKedTZJkt6fsmXL8ueff2bJdnxaRyAiHUXkdxHZJSKXpEoRKSgiX7jTV4lINV/GA7B//yk6dfqMXr2+4cSJGDp0qME333S3JGDypKQ6gj/++IMLFy4k1xE0aNCAtWvXXjTvnj17KFq0KKGhoTRo0IB169ZluP60EornuJT3tBcpUiT59Q033MCuXbs4evQo33zzDXfd5dyinZiYyIoVK9iwYQMbNmzg4MGDFyWBpH3zXPdLL73ErbfeypYtW/j2228vmua5TVXl+eefT173rl276NOnD0uXLmXx4sWsWLGCjRs30rRp01Tvxx8yZAhhYWGX/CWV61+OcuXKcejQIcBJcGXLOk3XV6pUiQMHDiTPFxkZSYUKTudWsbGxFCpU6LK36clniUBEgoEJwO1AfeA+EamfYrY+QJSq1gRGA2/4Kp7ERGHi8uY0aDCR+fN3UaJECNOmdeG//+1BtWrFfbVZY3KEYsWKMW7cON5++23i4uLo0aMHP//8M4sXLwacK4fBgwfzzDPPAPD000/z+uuvs2PHDsD5YX733XcvWW+HDh0YP3588nBS0VC5cuXYvn17ctFPWkSEv/3tbwwdOpR69eoln32nXO+GDRsuWbZevXoXlb2fOnWKihUrAjBt2rQ0t3nbbbcxderU5DqMgwcPcuTIEU6dOkWJEiUoXLgwv/32GytXrkx1+dGjRycnEc+/yy0WAoiIiOCTTz4B4JNPPqFLly7J46dPn46qsnLlSooVK5ZchLRjx46LisauhC+vCK4DdqnqHlW9AMwEuqSYpwvwift6FtBWfHRqfiq2ICMX3cKZMxe4++56bNv2OA8+GGZXAiZgNG3alCZNmjBz5kwKFSrEnDlzeO2116hTpw6NGjWiefPmybc4Nm7cmDFjxnDfffdRr149GjZsmHzG6mn48OFERUXRsGFDmjRpklx2PmrUKO68807atGmT/MOVlm7duvHpp58mFwsBjBs3jrVr19K4cWPq16/PBx98cMlydevW5dSpU0RHRwPwzDPP8Pzzz9OyZUsSEhLS3F6HDh24//77ueGGG2jUqBH33HMP0dHRdOzYkfj4eBo3bsxLL71EixYtMj6oXqhWrRpDhw5l2rRpVKpUKbnI65FHHkm+KnvuuedYtGgRtWrVYtGiRclJ5Y477uCaa66hZs2a9O3bN/muLoAlS5bQqVOnLIlRUiuDypIVi9wDdFTVR9zhnsD1qjrQY54t7jyR7vBud55jKdbVD+gHUKVKlWZ//PFH5gN6R/h2a20udJrN3XenvDAxJutt376devXq+TuMPG306NGEhobm6GcJfKVVq1bMmTMn1XqZ1D57IrJOVcNTW5cvK4tTO9VOmXW8mQdVnQRMAggPD7+8zPWU0vmyFjTG5FQDBgzg//7v//wdRrY7evQoQ4cO9apy3hu+LBqKBCp7DFcCUlZxJ88jIvmAYsAJH8ZkjMlDQkJC6Nmzp7/DyHZlypSha9euWbY+XyaCNUAtEakuIgWA7sDcFPPMBR50X98D/KC+Kqsyxg/s42yy2+V85nyWCFQ1HhgILAC2A1+q6lYReUVEItzZpgClRGQXMBS4/Gp3Y3KYkJAQjh8/bsnAZJuk/ghCQkIytZzPKot9JTw8XFPe/2xMTmQ9lBl/SKuHMn9VFhsT0PLnz5+pXqKM8RdrfdQYYwKcJQJjjAlwlgiMMSbA5brKYhE5ClzGo8UAlAaOZThX3mL7HBhsnwPDlexzVVUtk9qEXJcIroSIrE2r1jyvsn0ODLbPgcFX+2xFQ8YYE+AsERhjTIALtEQwyd8B+IHtc2CwfQ4MPtnngKojMMYYc6lAuyIwxhiTgiUCY4wJcHkyEYhIRxH5XUR2icglLZqKSEER+cKdvkpEqmV/lFnLi30eKiLbRGSTiHwvIlX9EWdWymifPea7R0RURHL9rYbe7LOI3Ou+11tF5LPsjjGrefHZriIiS0Rkvfv5vsMfcWYVEZkqIkfcHhxTmy4iMs49HptE5Nor3qiq5qk/IBjYDVwDFAA2AvVTzPMY8IH7ujvwhb/jzoZ9vhUo7L4eEAj77M4XCiwDVgLh/o47G97nWsB6oIQ7XNbfcWfDPk8CBriv6wP7/B33Fe5zK+BaYEsa0+8A5uP08NgCWHWl28yLVwTXAbtUdY+qXgBmAl1SzNMF+MR9PQtoK7m7F/sM91lVl6jqOXdwJU6PcbmZN+8zwKvAm0BeaAvam33uC0xQ1SgAVT2SzTFmNW/2WYGr3NfFuLQnxFxFVZeRfk+NXYDp6lgJFBeR8leyzbyYCCoCBzyGI91xqc6jTgc6p4BS2RKdb3izz5764JxR5GYZ7rOINAUqq+p32RmYD3nzPtcGaovIchFZKSIdsy063/Bmn18GHhCRSGAeMCh7QvObzH7fM5QX+yNI7cw+5T2y3syTm3i9PyLyABAO3OLTiHwv3X0WkSBgNNA7uwLKBt68z/lwioda41z1/SQiDVX1pI9j8xVv9vk+YJqqviMiNwD/dvc50ffh+UWW/37lxSuCSKCyx3AlLr1UTJ5HRPLhXE6mdymW03mzz4hIO+BFIEJVz2dTbL6S0T6HAg2BpSKyD6csdW4urzD29rM9R1XjVHUv8DtOYsitvNnnPsCXAKq6AgjBaZwtr/Lq+54ZeTERrAFqiUh1ESmAUxk8N8U8c4EH3df3AD+oWwuTS2W4z24xyYc4SSC3lxtDBvusqqdUtbSqVlPVajj1IhGqmpv7OfXms/0Nzo0BiEhpnKKiPdkaZdbyZp/3A20BRKQeTiI4mq1RZq+5QC/37qEWwClVPXQlK8xzRUOqGi8iA4EFOHccTFXVrSLyCrBWVecCU3AuH3fhXAl091/EV87LfX4LKAr8n1svvl9VI/wW9BXycp/zFC/3eQHQQUS2AQnA06p63H9RXxkv9/kp4CMRGYJTRNI7N5/YicjnOEV7pd16jxFAfgBV/QCnHuQOYBdwDnjoireZi4+XMcaYLJAXi4aMMcZkgiUCY4wJcJYIjDEmwFkiMMaYAGeJwBhjApwlApPjiEiCiGzw+KuWzrzV0mqlMZPbXOq2cLnRbZ6hzmWso7+I9HJf9xaRCh7TJotI/SyOc42IhHmxzJMiUvhKt23yLksEJieKUdUwj7992bTdHqraBKdBwrcyu7CqfqCq093B3kAFj2mPqOq2LInyf3FOxLs4nwQsEZg0WSIwuYJ75v+TiPzq/t2YyjwNRGS1exWxSURqueMf8Bj/oYgEZ7C5ZUBNd9m2bjv3m9124gu640fJ//p3eNsd97KIDBORe3Dac5rhbrOQeyYfLiIDRORNj5h7i8h7lxnnCjwaGxOR90VkrTj9EIx0xw3GSUhLRGSJO66DiKxwj+P/iUjRDLZj8jhLBCYnKuRRLDTbHXcEaK+q1wLdgHGpLNcfGKuqYTg/xJFukwPdgJbu+ASgRwbb7wxsFpEQYBrQTVUb4TyJP0BESgJ/AxqoamPgNc+FVXUWsBbnzD1MVWM8Js8C7vIY7gZ8cZlxdsRpUiLJi6oaDjQGbhGRxqo6DqcdmltV9Va32YnhQDv3WK4FhmawHZPH5bkmJkyeEOP+GHrKD4x3y8QTcNrQSWkF8KKIVAK+VtWdItIWaAascZvWKISTVFIzQ0RigH04TRnXAfaq6g53+ifA48B4nP4NJovIfwCvm7lW1aMissdtI2anu43l7nozE2cRnCYXPHunuldE+uF8r8vjdNKyKcWyLdzxy93tFMA5biaAWSIwucUQ4C+gCc6V7CUdzajqZyKyCugELBCRR3Ca7P1EVZ/3Yhs9PBulE5FU+6hw27+5Dqehs+7AQKBNJvblC+Be4DdgtqqqOL/KXseJ01PXKGACcJeIVAeGAc1VNUpEpuE0vpaSAItU9b5MxGvyOCsaMrlFMeCQ28Z8T5yz4YuIyDXAHrc4ZC5OEcn3wD0iUtadp6R431/zb0A1EanpDvcEfnTL1Iup6jycitjU7tyJxmkKOzVfA11x2tH/wh2XqThVNQ6niKeFW6x0FXAWOCUi5YDb04hlJdAyaZ9EpLCIpHZ1ZQKIJQKTW0wEHhSRlTjFQmdTmacbsEVENgB1cbrz24bzg7lQRDYBi3CKTTKkqrE4LTv+n4hsBhKBD3B+VL9z1/cjztVKStOAD5Iqi1OsNwrYBlRV1dXuuEzH6dY9vAMMU9WNOH0VbwWm4hQ3JZkEzBeRJap6FOeOps/d7azEOVYmgFnro8YYE+DsisAYYwKcJQJjjAlwlgiMMSbAWSIwxpgAZ4nAGGMCnCUCY4wJcJYIjDEmwP0/YoEB5trusCUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.svm import SVC\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "iris = load_iris()\n",
    "x = iris.data[:,:2]\n",
    "y = iris.target\n",
    "#为了构造一个二分类，将1 2类划分成非0类  \n",
    "y[np.where(y>0)] = 1\n",
    "x_train,x_test,y_train,y_test = train_test_split(x,y,test_size= 0.3)\n",
    "\n",
    "\n",
    "#用单个模型 做ROC曲线\n",
    "model =SVC(kernel='linear',probability=True)\n",
    "model.fit(x_train,y_train)\n",
    "test_pred = model.predict_proba(x_test)[:,1]\n",
    "\n",
    "test_auc = roc_auc_score(y_test,test_pred)\n",
    "fpr,tpr,ther = roc_curve(y_test,test_pred)\n",
    "\n",
    "plt.figure()\n",
    "lw = 2\n",
    "\n",
    "plt.plot(fpr, tpr, color='darkorange',\n",
    "         lw=lw, label='ROC curve (area = %0.2f)' % test_auc)\n",
    "\n",
    "plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n",
    "# plt.xlim([0.0, 1.0])\n",
    "# plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver operating characteristic example')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
